Stacked Auto Encoder Based Deep Reinforcement Learning for Online Resource Scheduling in Large-Scale MEC Networks

An online resource scheduling framework is proposed for minimizing the sum of weighted task latency for all the Internet-of-Things (IoT) users, by optimizing offloading decision, transmission power, and resource allocation in the large-scale mobile-edge computing (MEC) system. Toward this end, a deep reinforcement learning (DRL)-based solution is proposed, which includes the following components. First, a related and regularized stacked autoencoder (2r-SAE) with unsupervised learning is applied to perform data compression and representation for high-dimensional channel quality information (CQI) data, which can reduce the state space for DRL. Second, we present an adaptive simulated annealing approach (ASA) as the action search method of DRL, in which an adaptive h -mutation is used to guide the search direction and an adaptive iteration is proposed to enhance the search efficiency during the DRL process. Third, a preserved and prioritized experience replay (2p-ER) is introduced to assist the DRL to train the policy network and find the optimal offloading policy. The numerical results are provided to demonstrate that the proposed algorithm can achieve near-optimal performance while significantly decreasing the computational time compared with existing benchmarks

Offline Deep Reinforcement Learning and Off-Policy Evaluation for Personalized Basal Insulin Control in Type 1 Diabetes

Recent advancements in hybrid closed-loop systems, also known as the artificial pancreas (AP), have been shown to optimize glucose control and reduce the self-management burdens for people living with type 1 diabetes (T1D). AP systems can adjust the basal infusion rates of insulin pumps, facilitated by real-time communication with continuous glucose monitoring. Empowered by deep neural networks, deep reinforcement learning (DRL) has introduced new paradigms of basal insulin control algorithms. However, all the existing DRL-based AP controllers require a large number of random online interactions between the agent and environment. While this can be validated in T1D simulators, it becomes impractical in real-world clinical settings. To this end, we propose an offline DRL framework that can develop and validate models for basal insulin control entirely offline. It comprises a DRL model based on the twin delayed deep deterministic policy gradient and behavior cloning, as well as off-policy evaluation (OPE) using fitted Q evaluation. We evaluated the proposed framework on an in silico dataset containing 10 virtual adults and 10 virtual adolescents, generated by the UVA/Padova T1D simulator, and the OhioT1DM dataset, a clinical dataset with 12 real T1D subjects. The performance on the in silico dataset shows that the offline DRL algorithm significantly increased time in range while reducing time below range and time above range for both adult and adolescent groups. The high Spearman's rank correlation coefficients between actual and estimated policy values indicate the accurate estimation made by the OPE. Then, we used the OPE to estimate model performance on the clinical dataset, where a notable increase in policy values was observed for each subject. The results demonstrate that the proposed framework is a viable and safe method for improving personalized basal insulin control in T1D

Structured Compressed Sensing Using Deterministic Sequences

The problem of estimating sparse signals based on incomplete set of noiseless or noisy measurements has been investigated for a long time from different perspec- tives. In this dissertation, after the review of the theory of compressed sensing (CS) and existing structured sensing matrices, a new class of convolutional sensing matri- ces based on deterministic sequences are developed in the first part. The proposed matrices can achieve a near optimal bound with O(K log(N)) measurements for non-uniform recovery. Not only are they able to approximate compressible signals in the time domain, but they can also recover sparse signals in the frequency and discrete cosine transform domain. The candidates of the deterministic sequences include maximum length sequence (or called m-sequence), Golay's complementary sequence and Legendre sequence etc., which will be investigated respectively. In the second part, Golay-paired Hadamard matrices are introduced as structured sensing matrices, which are constructed from the Hadamard matrix, followed by diagonal Golay sequences. The properties and performances are analyzed in the following. Their strong structures ensure special isometry properties, and make them be easier applicable to hardware potentially. Finally, we exploit novel CS principles successfully in a few real applications, including radar imaging and dis- tributed source coding. The performance and the effectiveness of each scenario are verified in both theory and simulations

Surface defects repairing of sprayed Ca-P coating by the microwave-hydrothermal method

The increasing interest in decreasing the surface defects of sprayed Ca-P coating deposited on carbon/carbon (C/C) composites to enhance the bonding strength, bioactivity and corrosion resistance of the coating is justified by the growing evidence of its beneficial effect on the bone replacement fields. Microwave-hydrothermal (MH) method detailed in the previous study is successfully used to reduce the above coating defects and the MH mechanism is well studied here. Hence, five different treatment reagents involving calcium and phosphorus solution, sulfuric acid (H2SO4) solution, ammonium hydroxide (NH3·H2O) solution, only Ca2+ solution and deionized water are selected as the precursor solution. The surface, cross-sectional morphologies, phase and composition of the coatings are characterized by the scanning electron microscopy (SEM), X-ray diffraction (XRD), Fourier transform infrared spectroscopy (FTIR), microscopy Raman spectroscopy and X-ray photoelectron spectroscopy (XPS) spectra. Elastic modulus and coating hardness are measured by nanoindentation. Results reveal that the presence of calcium and phosphorus ions, as well as the H2SO4 in the precursor solution during the MH process, have a positive influence on the reduction of sprayed Ca-P coating surface defects. However, the coating treated by other three solutions cannot produce new phases on the basis of sprayed Ca-P coating and the surface defects of it are not decreased. Nevertheless, the elastic modulus and hardness of the coating treated by H2SO4 solution are very weak. MH treated coating by calcium and phosphorus ions in the precursor solution and in NH3·H2O solution, only Ca2+ solution and deionized water own the similar elastic modulus and hardness to that of the sprayed Ca-P coating. To conclude, in the MH process, the surface defects of the sprayed Ca-P coating are only lowered in calcium and phosphorus precursor solution and the coating strength is not dropped, which demonstrates the promoting mechanism of MH process